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Preface 


Autonomous vehicles will not be viable for real-world use on public roads 
unless we can make them acceptably safe. Not perfectly safe to the point of 
zero crashes — although that is a worthy goal. Rather, acceptably safe will do 
for commercial deployment, with a hope of providing a substantive 
improvement over the current mishap rate of human drivers. 

An acceptably safe outcome for autonomous vehicle (AV) deployment is 
not a foregone conclusion. Safety engineering does not happen all by itself, 
not even if super-smart engineers have the best intentions to create safe AVs. 
Simply being smart will not ensure all aspects of AV safety are covered any 
more than being smart will necessarily instill the skills and experience 
needed to make a safe aircraft. Achieving a safe outcome requires having 
strong safety engineering skills, as well as applying lessons learned across 
many domains in how to create safe systems. 

Creating a safe system design has always required tremendous attention to 
detail. That in turn involves the use of specific safety engineering approaches 
such as hazard analysis, risk mitigation, and careful implementation of 
redundancy architectures. Following an industry-created safety standard 
helps ensure that the right approaches are used in the right way to achieve 
acceptable safety. 

Because of the novelty of machine learning technology and lack of a 
human driver to display an approximation of common sense, autonomous 
vehicles present significantly different and dramatically more challenging 
issues for ensuring safety than traditional vehicles. Different companies are 
trying different approaches tuned for different applications and different 
implementation architectures. We have not yet arrived at a fixed design 
approach for building a safe AV. Nonetheless, such vehicles are deployed on 
public roads, and safety remains a pressing question. 

The industry consensus at this point seems to be that safety will not be 
ensured by following a building code-style recipe for how to build an AV. 
Maybe that will happen someday, but not today. Rather, the industry has 
converged on the concept of a safety case as a way to argue that an AV is 
acceptably safe for its intended operations. The idea of a safety case for cars 
is not a new one. The decade-old ISO 26262 automotive functional safety 
standard requires a safety case. 

ANSI/UL 4600 extends the safety case approach to its logical conclusion 
for AVs, resulting in the most comprehensive standard for autonomous 
vehicle safety currently available. It describes how to assess that the AV’s 
safety case includes everything it should, to support a credible claim of 
acceptable safety. 

This book covers the background of the standard, how the standard is 
structured, key terminology, and a clause-by-clause summary of the standard. 
It is not a detailed restatement, but rather a high-level overview. Ideally, the 
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reader will go through this book, get the big picture, and then be ready to 
dive into the details of the standard itself. 

To keep things concise, this is a guided tour of the standard rather than an 
in-depth text on system safety. The style of writing is intended to be a 
descriptive narrative rather than an academic text. This should make concepts 
more accessible to those who are not safety engineering experts. (Those 
looking for the humorous footnote style seen in my previous book on “how 
safe is safe enough” will be disappointed. This is more of a just-the-facts 
guided tour.) 

If you are familiar with system safety concepts and functional safety, 
especially in the automotive industry, you might find yourself nodding along 
as you run down through the topics. If so, that’s great, because it means 
you’ re getting the big picture in mind as preparation to dive into the standard. 

If you hit a chapter on a topic you’ve not dealt with before, that is a great 
opportunity to expand your breadth in system safety before diving into the 
details of that part of the standard. Most chapters have a reference section 
with places to get started if a topic is new to you. If you are new to safety 
engineering for AVs in general, chapter 1 lists some getting-started 
resources. 

Creating UL 4600 has been quite a journey. I personally wrote the 
proposed text (200+ pages of it) of the initial draft that kicked things off. 
After submitting that draft, we followed an ANSI-conformant consensus 
process to ensure robust engagement with stakeholders. Hundreds of 
comments (many hundreds) arrived from all over the world. Suggestions, and 
sometimes complaints, were resolved. That feedback improved clarity, added 
essential elements, and resolved controversy. 

I have remained closely involved with the revision process for each 
edition via submitting change proposals, performing technical reviews, and 
commenting on proposed changes. But by no means am I the only one at 
work on this standard. 

As is appropriate for an industry standard, the entire revision and approval 
process is public. I get one vote out of 30-40 allocated to members of the 
Standards Technical Panel (STP) voting committee for eventual approval of 
each edition of the standard. The issued standard represents the results of an 
accredited industry standard consensus process that reflects inputs from 
vehicle makers, component suppliers, regulators, consumers, assessment 
organizations, safety researchers, and more. 

From time to time some industry politics have come into play — as they do 
with every standard. But a delightful thing about his process has been that the 
participants were overwhelmingly not there for the politics, but rather to get 
the job done. Many in the industry doubted that UL 4600 could be issued on 
our stretch-goal timeline of about a year from proposal to issued standard, 
but indeed that is how it turned out. It could never have been done without 
the common efforts, willingness to have frank discussions, and helpful 
contributions of so many Standards Technical Panel members and other 
stakeholders. Thank you so much to everyone who contributed! 
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If you wonder how I came to know enough about such a wide variety of 
topics to put into the draft proposal, chapter 18 has a brief bio. Suffice it to 
say that I’ve had a really broad range of experiences across many different 
industries, and seen an awful lot of stuff that is relevant to a standard like 
this. That includes doing hundreds of design reviews for products and 
components not only automotive, but also rail, industrial controls, building 
automation, power systems, and even a bit of work on aviation control 
networking safety. Much of UL 4600 falls into the bins of “don’t make this 
mistake because that turned out badly for someone else,” “this is how safety 
is done in other industries in addition to automotive” and “did you think of 
that?” 

While many stakeholders made valuable contributions, there are a few 
special contributors I want to thank in particular. Deb Prince wrangled 
everyone (including me) through the standard process and supported my 
sometimes unconventional approaches. Jackie Erickson provided invaluable 
contributions to stakeholder outreach and messaging, especially with 
regulators and media. Heather Sakellariou did the heavy lifting on logistics, 
editing, comment management, and production for the standard. Uma Ferrell 
provided pivotal feedback on the early outline as well as lessons drawn from 
her extensive aviation safety experience. Thanks also to Frank Fratrik, Jason 
Smith, and Mahmood Tabaddor for their contributions to the drafting 
process. Jack Weast, Rafael Zalman, Finch Fulton, Nat Beuse, Junko 
Yoshida, Roger Cohen, Aaron Kane, and Chuck Weinstock also provided 
particularly important discussions, support, and other contributions. 

Nothing is ever perfect, and everything can be improved. But fortunately, 
both this book and the UL 4600 standard itself can be updated with 
comparatively little pain. If you see something that should be fixed, please let 
me know via an e-mail to AVSafety@Koopman.us 


Meanwhile, happy reading! 


Philip Koopman 
Pittsburgh, PA, November 2022. 


Introduction 1 


1. Introduction 
Welcome to the world of UL 4600! 


This book boils a lengthy, dense, and complex standard down as much as 
can be done while still being a comprehensive treatment for something that is 
— well — lengthy, dense, and complex. 

UL 4600 takes a detailed, thorough approach because its purpose is to 
make sure nothing important gets left out of the safety case for an 
autonomous vehicle (AV). AVs are incredibly complex, so there is a lot of 
ground to cover. 

This book, in contrast, uses a more narrative approach. General themes, 
ideas, and considerations are described, often in an order that flows better 
from a narrative point of view. While most chapters correspond to the 
sections of UL 4600, this book’s subsections do not necessarily follow the 
exact flow of the standard’s subsections. Rather, this book follows an order 
better suited to telling the high-level story of each corresponding clause 
(chapter) in UL 4600. 

Not every detail can be in this book. Rather, the main ideas are discussed 
in a general sense. Think of this book as a way to understand the main 
themes and get an orientation to what is going on, without getting bogged 
down in the mechanics of the standard itself. After all, if you really want the 
gory details, that is what the standard is for. 


1.1. Quick tour 


Here is a quick tour of the rest of this book: 

e Chapter 2 covers the history and scope of UL 4600. Briefly, it deals with 
how to know that an autonomous vehicle (AV) safety case has what it 
needs to ensure that an AV will be acceptably safe. There is also a 
Frequently Asked Questions section that answers common questions and 
clarifies some common misconceptions regarding UL 4600. 


e Chapter 3 describes the structure of UL 4600, which emphasizes the use 
of “prompt elements” to help remind both safety engineers and safety 
case assessors what should be addressed by the safety case. It is 
important to be oriented to this approach before diving into the standard 
itself. 

e Chapter 4 covers key terminology and concepts. Every standard has 
some defined terms with nuances not necessarily easy to interpret 
without a little introductory guidance. Read this chapter if you want to 
know what UL 4600 might mean by “acceptable,” “item,” and “argue,” 
among other terms. 
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e Chapters 5-17 cover the corresponding clauses in UL 4600, ranging from 
safety cases in chapter 5 to the assessment process in chapter 17. 


e Chapter 18 has some pointers to additional information that might prove 
useful. 


1.2. Resources 


e UL 4600 information launch page: 
https://users.ece.cmu.edu/~koopman/ul4600/index.html 

e Video tutorial on UL 4600 (23 minutes): 

o YouTube version: https://youtu.be/Zx VMX8SjPvw 
o  Archive.org version: https://archive.org/details/L109-ul-4600 
e Video tutorial series on AV safety to provide background 
© https://users.ece.cmu.edu/~koopman/lectures/index.html#av 
(includes slides, YouTube videos, and archive.org mirrors) 

e A graduate-level course on embedded system and software safety taught 
by the author at Carnegie Mellon University, with all lecture videos 
freely available online: https://course.ece.cmu.edu/~ece642/ 

e Some historical notes on the evolution of UL 4600: 
https://www.eetimes.com/safe-autonomy-ul-4600-and-how-it-grew/ 


